Multi-modal tone-mapping of images

ABSTRACT

A system for multi-modal mapping of images is described. Embodiments are described where the image mapping system is used for visualizing high dynamic range images such as medical images, satellite images, high dynamic range photographs and the like and also for compressing such images. In examples, high bit-depth images are tone-mapped for display on equipment of lower bit-depth without loss of detail. In embodiments, the image mapping system computes statistics describing an input image and fits a multi-modal model to those statistics efficiently. In embodiments, the multi-modal model is a Gaussian mixture model and a plurality of sigmoid functions corresponding to the multi-modal model are obtained. In an embodiment the sigmoid functions are added to form a tone-mapping function which is used to transform a high bit-depth image such as 16 or 12 bits per pixel to a low bit-depth image such as 8 bits per pixel.

BACKGROUND

Image mapping is carried out for many purposes such as imagecompression, contrast enhancement, and for enabling images captured withcapture devices of particular types to be displayed using displaydevices of different capabilities. For example, in the field of medicalimaging (or in other fields such as professional photography, roboticimaging systems, high dynamic range photography, depth cameras, capturedevices often produce 16-bit images where for example, each pixel may beone of 65,536 levels of grey (in the case of a greyscale image). Otherimage capture devices may produce 12-bit images or 32-bit imagesdepending on the image capture device. The term “bit-depth” is used torefer to the number of bits available per pixel at an image capture ordisplay device.

Where images have been captured with high bit-depth devices it is oftenrequired to reduce the bit-depth to enable the captured images to bedisplayed on a display device with lower bit-depth. This is difficult toachieve whilst preserving as much information as possible, so as not tolose the original dynamic range captured in the high bit-depth device.This is important for many types of images and particularly so in thefield of medical imaging, where images often have particularly highdynamic range and where it is required to visualize fine details inimages and remove noise as far as possible in order to make accuratemedical diagnoses.

Dynamic range of an image may be thought of as the ratio between theintensities of the brightest and darkest recordable parts of that image.Tone-mapping functions are typically used to compress the dynamic rangeof an image to allow more detail in the original image to be visualizedon a display whilst preferably preserving the “natural look” of theimage. Improved tone-mapping systems are required which may produce morerealistic, useful results in a computationally inexpensive, fast androbust manner.

Where images are captured at devices with relatively high bit-depth, itis often required to compress those captured images to reduce their sizefor storage and/or transmission. Image compression is difficult toachieve in a manner which is computationally inexpensive, fast, whichdoes not produce visible artefacts and which is reversible (that is, theoriginal image can be obtained from the compressed image without loss ofquality).

Previous approaches for mapping images of one bit-depth to anotherbit-depth have included histogram equalization, linear mappings andgamma mappings. Linear mappings and gamma mappings are straightforwardtechniques but which are also very limited in the quality of resultsthey give.

Histogram equalization tone-mapping processes typically involve takingthe cumulative histogram of an image to be tone-mapped. The cumulativehistogram is then normalized to 255 (in the case that the outputbit-depth is 8 bits) and the normalized cumulative histogram is thenused as a mapping function to transform the original image to therequired bit-depth. However, histogram equalization processes are oftenfound to be very aggressive and as a result fine details in images arelost. Artefacts may also be introduced such as gradient reversal andquantization or banding artefacts.

Local histogram equalization tone-mapping processes are also known.These are sometimes referred to as adaptive histogram equalizingtechniques. They involve applying different transforms to equalize thehistograms of sub-regions of an image. These approaches are typicallyhighly computationally intensive and difficult to implement in real-timeapplications. Noise and artefacts may also be introduced for example,because a rectangular window is typically used around each pixel forhistogram equalization in this window. The resulting transform is thennot spatially smooth.

Previously, a single sigmoid function has been used as a tone-mappingfunction, with the sigmoid function determined from original imagestatistics and taking perceptual preference guidelines into account.Taking subjective preferences into account allows the image to look aspleasing as possible to the viewer. This is desirable in consumerimaging and commercial photography. However, this approach is notsuitable for medical imaging applications, satellite imaging, archivingand the like where the goal is information preservation. In such casesit is required to preserve or enhance details at all regions andluminance levels of an image, not just those suited to the human visualsystem.

The embodiments described below are not limited to implementations whichsolve any or all of the noted disadvantages of known image mappingsystems.

SUMMARY

The following presents a simplified summary of the disclosure in orderto provide a basic understanding to the reader. This summary is not anextensive overview of the disclosure and it does not identifykey/critical elements of the invention or delineate the scope of theinvention. Its sole purpose is to present some concepts disclosed hereinin a simplified form as a prelude to the more detailed description thatis presented later.

A system for multi-modal mapping of images is described. Embodiments aredescribed where the image mapping system is used for visualizing highdynamic range images such as medical images, satellite images, highdynamic range photographs and the like and also for compressing suchimages. In examples, high bit-depth images are tone-mapped for displayon equipment of lower bit-depth without loss of detail. In embodiments,the image mapping system computes statistics describing an input imageand fits a multi-modal model to those statistics efficiently. Inembodiments, the multi-modal model is a Gaussian mixture model and aplurality of sigmoid functions corresponding to the multi-modal modelare obtained. In an embodiment the sigmoid functions are added to form atone-mapping function which is used to transform a high bit-depth imagesuch as 16 or 12 bits per pixel to a low bit-depth image such as 8 bitsper pixel.

Many of the attendant features will be more readily appreciated as thesame becomes better understood by reference to the following detaileddescription considered in connection with the accompanying drawings.

DESCRIPTION OF THE DRAWINGS

The present description will be better understood from the followingdetailed description read in light of the accompanying drawings,wherein:

FIG. 1 a is a schematic diagram of a cumulative histogram of a highdynamic range image;

FIG. 1 b is a schematic diagram of a sigmoid tone mapping function;

FIG. 2 is a schematic diagram of a computational tomography (CT) scanimage and an associated cumulative histogram of that scan image which ismulti-modal;

FIG. 3 is a flow diagram of a method of image mapping;

FIG. 4 a is a schematic diagram of a cumulative histogram of a highdynamic range image and showing a schematic Gaussian mixture modelsuperimposed on the histogram;

FIG. 4 b is a schematic diagram of a tone-mapping function formed usingthe Gaussian mixture model of FIG. 4 a;

FIG. 5 is a flow diagram of a method of determining a Gaussian mixturemodel;

FIG. 6 is a flow diagram of a method of dynamically tone-mappingsub-regions of an image;

FIG. 7 a is a schematic diagram of apparatus arranged to compress,transmit and decompress a high bit-depth image;

FIG. 7 b is a schematic diagram of another apparatus arranged tocompress, transmit and decompress a high bit-depth image;

FIG. 8 is a schematic diagram of an image mapping apparatus;

FIG. 9 illustrates an exemplary computing-based device in whichembodiments of an image mapping system may be implemented.

Like reference numerals are used to designate like parts in theaccompanying drawings.

DETAILED DESCRIPTION

The detailed description provided below in connection with the appendeddrawings is intended as a description of the present examples and is notintended to represent the only forms in which the present example may beconstructed or utilized. The description sets forth the functions of theexample and the sequence of steps for constructing and operating theexample. However, the same or equivalent functions and sequences may beaccomplished by different examples.

Although the present examples are described and illustrated herein asbeing implemented in a medical imaging system, the system described isprovided as an example and not a limitation. As those skilled in the artwill appreciate, the present examples are suitable for application in avariety of different types of imaging systems.

FIG. 1 a is a schematic diagram of a cumulative histogram 100 of a highdynamic range image. FIG. 1 b is a schematic diagram of a single sigmoidfunction 101 which may be used as a tone-mapping function for the imagewhich produced the histogram of FIG. 1 a. For a given pixel in the highdynamic range input image, the intensity value is used to look-up thecorresponding low dynamic range pixel value using the sigmoid function.This process is satisfactory in the case that the cumulative histogramof the input image is uni-modal as in the example of FIG. 1 a. However,this is not the case for many types of high dynamic range images such asmedical images, satellite images, dental images, high dynamic rangephotographs and other images of scenes or items where there are manydistinct ranges of pixel intensities. This is explained in more detailwith reference to FIG. 2.

FIG. 2 shows a schematic diagram of a CT scan and an associatedcumulative image histogram. The CT scan is of a human body comprisingmuscle 201, bone 200 and lungs 202. The parts of the image representingbone tend to have pixel intensities in a high range 203 as shown in thehistogram. The parts of the image representing muscle tend to producepixel intensities in a different range 204 and the parts of the imagerepresenting lungs 205 produce pixel intensities in yet another range205. These three different ranges in the histogram are not adequatelytaken into account by a single sigmoid tone-mapping function such asthat of FIG. 1 b. If such a tone-mapping function is used the resultsare poor because much of the detail of the lungs and bone regions arelost. This is problematic for applications where informationpreservation is key such as for medical diagnosis.

To address such problems the embodiments described herein use amulti-modal model which is able to take into account more than one modein the cumulative histogram of the input image. Multi-modal models aremore complex than single mode models (such as that of FIG. 1 b) and yetthe embodiments described herein provide an image mapping system whichmay be used in real time for practical applications such as inhospitals, dental surgeries, weather forecasting systems and the like.

FIG. 2 is a flow diagram of a method of image mapping. An input digitalimage is received 300 which is a high dynamic range image of any type. Acumulative histogram is computed of the input image 301. For example,for a grey scale image a histogram generator determines a bin size (e.g.a range of values such as 0-10, 11-20, 21-30, etc.) and counts thenumber of occurrences of pixel grey scale values in each bin. A graph ofthe bins against the frequency of occurrence in each bin gives agraphical representation of the histogram such as that illustrated inFIG. 2.

A processor is arranged to fit 302 a multi-modal model to the histogram.Any suitable type of multi-modal model may be used such as a Gaussianmixture model (GMM). The number of modes (n) that are required may be aconfigurable parameter 303 or may be determined automatically by theimage mapping system. For many practical applications, including CTscans and MRI (magnetic resonance imaging) scans it is found that usingonly two modes provides greatly improved results as compared withprevious tone-mapping processes and that increasing the number of modesto three or more is advantageous in some cases.

In an embodiment, an n-component GMM model is fitted to the histogram.Various different methods of computing the GMM model are possible asdiscussed below with reference to FIG. 5. GMM models are described indetail in Bishop. C. M Pattern Recognition and Machine Learning. 2006Springer which is incorporated herein by reference in its entirety.

Using the Gaussian mixture model n corresponding mapping sigmoidfunctions are computed 304 and summed 306 to obtain a multi-modaltone-mapping function (also called transformation function). Thebit-depth of the output display 305 may be taken into account. This isdescribed in more detail below with reference to FIG. 4.

Once the tone-mapping function is obtained it is used to transform 307the high dynamic range input image into an output image with therequired bit-depth. The output image is stored or displayed on a displaydevice such as a computer screen or printer 308.

FIG. 4 a is a schematic diagram of a cumulative histogram of a highdynamic range image. The histogram is multi-modal and a GMM model isfitted with 3 components represented by Gaussian distributions 400, 401and 402. Each Gaussian distribution has an associated mean p andstandard deviation sigma. The parameters of the Gaussian distributionsare used as known in the art to form a sigmoid function corresponding toeach Gaussian distribution. These sigmoid functions 403, 404, 405 areadded to form a tone-mapping function as illustrated in FIG. 4 b. Theresulting tone-mapping function may be adjusted if necessary to ensurethat it begins at the origin in order that black pixels from the HDRimage map to black pixels in the output image. This ensures that thefull range of available bit-depth in the output is made use of.

The step of fitting the multi-modal model is relatively complex comparedwith other stages of the process and compared with previous approacheswhich have used uni-modal models. Various different methods of fittingthe multi-modal model may be used. For example, an n-component GMM model500 may be fitted using an expectation maximization logic provided inthe image mapping system. Expectation maximization is explained indetail in Bishop. C. M Pattern Recognition and Machine Learning. 2006Springer reference above. Another approach is to use a K-meansclustering logic provided in the image mapping system. K-meansclustering is also explained in detail in Bishop. C. M PatternRecognition and Machine Learning. 2006 Springer which is referencedabove. In order to provide a particularly fast implementation, anembodiment uses 1D K-means clustering in an efficient manner. In thisembodiment image data to be processed is received in the form of integervalues. For non-integer values these are first quantized and thentreated as integers. The minimum and maximum values in the integer dataset are determined and a histogram of the integer values is created.Cluster center positions are initialized to be equally spaced throughoutthe range of values. The following steps are then performed iterativelyuntil convergence:

For each of K clusters the 1D Voronoi interval is determined around thecurrent center position. For each histogram entry in the currentinterval, the number of values in that cluster are accumulated and thesum of those values is determined. After going through the wholeinterval the cluster's center of mass is re-computed as the sum dividedby the count. If the procedure produces de minimis change in the clustercenter positions then the procedure is terminated.

In some embodiments the image mapping process is dynamic in that thetone-mapping function is re-computed in real time for sub-regions of theinput image, for example, as a result of user input “zooming in” on aparticular part of the image. This is useful in many data visualizationapplications such as medical imaging where a medical doctor may need tosee more detail in a particular region of an image in order to make adiagnosis. The sub-region of the image may be specified by user input,such as via a graphical user interface, speech interface or otherinterface. Alternatively, the sub-region may be selected automaticallyby an image processing system.

For example as described with reference to FIG. 6, a tone-mapped imageis obtained using a first mapping function (which is multi-modal) 600and displayed. User input is received specifying a sub-region of theimage 601 and a second mapping function is computed in real-time 602.For example, the n-component GMM model is fitted to a cumulativehistogram obtained from the image sub-region. The display is thendynamically updated using the second mapping function 603 and optionallyusing a smoothing process to avoid an abrupt change in the display. Forexample, the second mapping function is applied to the image sub-regionand only the transformed sub-region displayed.

The processes described herein may also be used to enhance imagecompression. For example, previously a high bit-depth image 700 has beencompressed with a codec 701 suitable for high bit-depth images. Knowncompression algorithms for high bit-depth images are complex and timeconsuming. Often these introduce artifacts to the image. Once compressedthe image is transmitted 702 for example, over the Internet or othercommunications network before being uncompressed at a DE-CODEC 703. Theresulting high bit-depth image 704 typically contains artifacts.

In an embodiment a high bit depth image 700 is tone mapped to a lowerbit depth (such as 8 bits per pixel (bpp)) 705 using any of theembodiments described herein. The tone-mapped image is then compressed706 using any suitable known 8-bpp image compression algorithm. Thisresults in a higher quality compressed image because the 8 bpp range isutilized optimally thanks to the non-linear tone-mapping. Furthermore,existing 8-bpp compression algorithms are considerably faster thancorresponding 16-bpp ones. The compressed image is transmitted 702 anddecompressed 707 using the known compression process and may then bedisplayed 708. For example, in the case of a medical image, a doctor mayreceive an email with the attached compressed image, decompress thatimage and display it at his or her PC. The doctor is then able tovisualize the tone-mapped image with the same quality as the tone-mappedimage before sending. It is also possible to reverse the tone-mappingprocess 709 to obtain a high-dynamic range image back. In this case, thetransmission comprises both the compressed image and the parameters ofthe multi-modal model. These are used to reverse the tone-mappingprocess 709 and provide the high bit-depth image without artifacts aswould have previously been introduced. That high bit-depth image maythen be used for further processing or other purposes as required.

FIG. 8 is a schematic diagram of an image mapping apparatus 805comprising a tone mapping processor 801 arranged to use a multi-modalmodel such as those described herein to map a source image 800 of highbit-depth to a display device 802 of lower bit-depth. The display devicemay be a printer, display screen or other display apparatus. The imagemapping apparatus 805 also comprises an image statistics processor 803which determined statistics describing the source image 800 for use increating a multi-modal model. The image statistics processor is arrangedto fit a multi-modal model to the image statistics and to store theresulting model 804 in memory.

FIG. 9 illustrates various components of an exemplary computing-baseddevice 900 which may be implemented as any form of a computing and/orelectronic device, and in which embodiments of an image mapping systemmay be implemented.

The computing-based device 900 comprises one or more inputs 906 whichare of any suitable type for receiving media content, Internet Protocol(IP) input, high dynamic range images, high bit-depth images and otherinput. The device also comprises communication interface 907 for exampleto connect the device to the Internet or other communications networkfor transmission of images and/or other data.

Computing-based device 900 also comprises one or more processors 901which may be microprocessors, controllers or any other suitable type ofprocessors for processing computing executable instructions to controlthe operation of the device in order to map high bit-depth images tolower bit-depth images whilst retaining dynamic range information.Platform software comprising an operating system 904 or any othersuitable platform software may be provided at the computing-based deviceto enable application software 903 to be executed on the device.

The computer executable instructions may be provided using anycomputer-readable media, such as memory 902. The memory is of anysuitable type such as random access memory (RAM), a disk storage deviceof any type such as a magnetic or optical storage device, a hard diskdrive, or a CD, DVD or other disc drive. Flash memory, EPROM or EEPROMmay also be used.

An output is also provided such as an audio and/or video output to adisplay system integral with or in communication with thecomputing-based device. A display interface 905 may provide a graphicaluser interface, or other user interface of any suitable type althoughthis is not essential.

The term ‘computer’ is used herein to refer to any device withprocessing capability such that it can execute instructions. Thoseskilled in the art will realize that such processing capabilities areincorporated into many different devices and therefore the term‘computer’ includes PCs, servers, mobile telephones, personal digitalassistants and many other devices.

The methods described herein may be performed by software in machinereadable form on a tangible storage medium. The software can be suitablefor execution on a parallel processor or a serial processor such thatthe method steps may be carried out in any suitable order, orsubstantially simultaneously.

This acknowledges that software can be a valuable, separately tradablecommodity. It is intended to encompass software, which runs on orcontrols “dumb” or standard hardware, to carry out the desiredfunctions. It is also intended to encompass software which “describes”or defines the configuration of hardware, such as HDL (hardwaredescription language) software, as is used for designing silicon chips,or for configuring universal programmable chips, to carry out desiredfunctions.

Those skilled in the art will realize that storage devices utilized tostore program instructions can be distributed across a network. Forexample, a remote computer may store an example of the process describedas software. A local or terminal computer may access the remote computerand download a part or all of the software to run the program.Alternatively, the local computer may download pieces of the software asneeded, or execute some software instructions at the local terminal andsome at the remote computer (or computer network). Those skilled in theart will also realize that by utilizing conventional techniques known tothose skilled in the art that all, or a portion of the softwareinstructions may be carried out by a dedicated circuit, such as a DSP,programmable logic array, or the like.

Any range or device value given herein may be extended or alteredwithout losing the effect sought, as will be apparent to the skilledperson.

It will be understood that the benefits and advantages described abovemay relate to one embodiment or may relate to several embodiments. Theembodiments are not limited to those that solve any or all of the statedproblems or those that have any or all of the stated benefits andadvantages. It will further be understood that reference to ‘an’ itemrefers to one or more of those items.

The steps of the methods described herein may be carried out in anysuitable order, or simultaneously where appropriate. Additionally,individual blocks may be deleted from any of the methods withoutdeparting from the spirit and scope of the subject matter describedherein. Aspects of any of the examples described above may be combinedwith aspects of any of the other examples described to form furtherexamples without losing the effect sought.

The term ‘comprising’ is used herein to mean including the method blocksor elements identified, but that such blocks or elements do not comprisean exclusive list and a method or apparatus may contain additionalblocks or elements.

It will be understood that the above description of a preferredembodiment is given by way of example only and that variousmodifications may be made by those skilled in the art. The abovespecification, examples and data provide a complete description of thestructure and use of exemplary embodiments of the invention. Althoughvarious embodiments of the invention have been described above with acertain degree of particularity, or with reference to one or moreindividual embodiments, those skilled in the art could make numerousalterations to the disclosed embodiments without departing from thespirit or scope of this invention.

1. An image mapping system comprising: an input arranged to receive atleast part of an input image of a first bit-depth; a processor arrangedto compute statistics describing the input image and to fit amulti-modal model to the statistics; the processor also being arrangedto determine a multi-modal tone-mapping function by determining aplurality of sigmoid functions, one for each mode of the multi-modalmodel; and a tone-mapping processor arranged to apply the tone-mappingfunction to the received input image to form an output image ofbit-depth different from the input image.
 2. An image mapping system asclaimed in claim 1 wherein the processor is arranged to compute acumulative histogram of the input image.
 3. An image mapping system asclaimed in claim 2 wherein the processor is arranged to fit ann-component Gaussian mixture model to the cumulative histogram.
 4. Animage mapping system as claimed in claim 1 wherein the processor isarranged to determine the tone-mapping function by adding the pluralityof sigmoid functions.
 5. An image mapping system as claimed in claim 1wherein the processor is arranged to fit the multi-modal model using aK-means clustering process.
 6. An image mapping system as claimed inclaim 1 wherein the input is further arranged to receive details of asub-region of the input image and wherein the processor is arranged tocompute second statistics describing the sub-region of the input imageand to fit a second multi-modal model to those second statistics.
 7. Animage mapping system as claimed in claim 6 wherein the processor is alsoarranged to determine a second multi-modal tone-mapping function usingthe second multi-modal model.
 8. An image mapping system as claimed inclaim 7 wherein the tone-mapping processor is arranged to apply thesecond tone-mapping function to the image sub-region.
 9. An imagemapping system as claimed in claim 1 which further comprises an imagecompression device arranged to take the output image as input.
 10. Animage mapping system as claimed in claim 1 which is arranged to operateon images of bit-depth greater than 8 bits per pixel.
 11. An imagemapping system as claimed in claim 1 which is arranged to operate onmedical images.
 12. An image mapping system as claimed in claim 1wherein the tone-mapping processor is also arranged to reverse thetone-mapping process in order to obtain the input image from the outputimage and the tone-mapping function without substantial loss ofinformation in the input image.
 13. A medical image mapping systemcomprising: an input arranged to receive at least part of an input imageof a first bit-depth; a processor arranged to compute statisticsdescribing the input image and to fit a multi-modal model to thestatistics; the processor also being arranged to determine a multi-modaltone-mapping function comprising a plurality of sigmoid functions, onefor each mode of the multi-modal model; and a tone-mapping processorarranged to apply the tone-mapping function to the received input imageto form an output image of bit-depth different from the input image. 14.An image mapping system as claimed in claim 13 wherein the processor isarranged to compute a cumulative histogram of the input image.
 15. Amethod of image mapping comprising: receiving at an image processingsystem at least part of an input image of a first bit-depth; computingstatistics describing the input image using a processor; fitting amulti-modal model to the statistics using the processor; determining amulti-modal tone-mapping function from the multi-modal model using theprocessor, wherein the determining comprises determining a plurality ofsigmoid functions, one for each mode of the multi-modal model; andapplying the tone-mapping function to the received input image using atone-mapping processor to form an output image of bit-depth differentfrom the input image.
 16. A method as claimed in claim 15 wherein thestep of computing the statistics comprises computing a cumulativehistogram of the input image.
 17. A method as claimed in claim 16wherein the step of fitting a multi-modal model comprises fitting ann-component Gaussian mixture model to the cumulative histogram.
 18. Amethod as claimed in claim 15 wherein the step of fitting a multi-modalmodel comprises fitting a bi-modal model.